{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 垃圾邮件分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "SPAM_PATH = os.path.join(\"datasets\",\"spam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "HAM_DIR = os.path.join(SPAM_PATH,\"easy_ham\") #正常邮件数据集路径\n",
    "SPAM_DIR = os.path.join(SPAM_PATH,\"spam\") #垃圾邮件数据集路径\n",
    "#获得所有文件的文件名\n",
    "ham_filenames = [name for name in sorted(os.listdir(HAM_DIR)) if len(name) > 20]\n",
    "spam_filenames = [name for name in sorted(os.listdir(SPAM_DIR)) if len(name) > 20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用python的\"email\"模块解析这些电子邮件（他可以处理邮件头，编码等）\n",
    "import email\n",
    "import email.policy\n",
    "\n",
    "def load_email(is_spam,filename,spam_path=SPAM_PATH):\n",
    "    directory = \"spam\" if is_spam else \"easy_ham\"\n",
    "    with open(os.path.join(spam_path,directory,filename),\"rb\") as f:\n",
    "        return email.parser.BytesParser(policy=email.policy.default).parse(f) #读取二进制字节数组，返回一个email对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Martin A posted:\n",
      "Tassos Papadopoulos, the Greek sculptor behind the plan, judged that the\n",
      " limestone of Mount Kerdylio, 70 miles east of Salonika and not far from the\n",
      " Mount Athos monastic community, was ideal for the patriotic sculpture. \n",
      " \n",
      " As well as Alexander's granite features, 240 ft high and 170 ft wide, a\n",
      " museum, a restored amphitheatre and car park for admiring crowds are\n",
      "planned\n",
      "---------------------\n",
      "So is this mountain limestone or granite?\n",
      "If it's limestone, it'll weather pretty fast.\n",
      "\n",
      "------------------------ Yahoo! Groups Sponsor ---------------------~-->\n",
      "4 DVDs Free +s&p Join Now\n",
      "http://us.click.yahoo.com/pt6YBB/NXiEAA/mG3HAA/7gSolB/TM\n",
      "---------------------------------------------------------------------~->\n",
      "\n",
      "To unsubscribe from this group, send an email to:\n",
      "forteana-unsubscribe@egroups.com\n",
      "\n",
      " \n",
      "\n",
      "Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/\n"
     ]
    }
   ],
   "source": [
    "#展示一个ham实例和一个spam实例，了解数据的外观\n",
    "ham_emails = [load_email(is_spam=False,filename=name) for name in ham_filenames] # 获取所有的正常邮件数据集\n",
    "spam_emails = [load_email(is_spam=True,filename=name) for name in spam_filenames] # 获取所有的垃圾邮件数据集\n",
    "print(ham_emails[1].get_content().strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "##################################################\n",
      "#                                                #\n",
      "#                 Adult Club                     #\n",
      "#           Offers FREE Membership               #\n",
      "#                                                #\n",
      "##################################################\n",
      "\n",
      ">>>>>  INSTANT ACCESS TO ALL SITES NOW\n",
      ">>>>>  Your User Name And Password is.\n",
      ">>>>>  User Name: zzzz@spamassassin.taint.org\n",
      ">>>>>  Password: 760382\n",
      "\n",
      "5 of the Best Adult Sites on the Internet for FREE!\n",
      "---------------------------------------\n",
      "NEWS 08/18/02\n",
      "With just over 2.9 Million Members that signed up for FREE, Last month there were 721,184 New\n",
      "Members. Are you one of them yet???\n",
      "---------------------------------------\n",
      "Our Membership FAQ\n",
      "\n",
      "Q. Why are you offering free access to 5 adult membership sites for free?\n",
      "A. I have advertisers that pay me for ad space so you don't have to pay for membership.\n",
      "\n",
      "Q. Is it true my membership is for life?\n",
      "A. Absolutely you'll never have to pay a cent the advertisers do.\n",
      "\n",
      "Q. Can I give my account to my friends and family?\n",
      "A. Yes, as long they are over the age of 18.\n",
      "\n",
      "Q. Do I have to sign up for all 5 membership sites?\n",
      "A. No just one to get access to all of them.\n",
      "\n",
      "Q. How do I get started?\n",
      "A. Click on one of the following links below to become a member.\n",
      "\n",
      "- These are multi million dollar operations with policies and rules.\n",
      "- Fill in the required info and they won't charge you for the Free pass!\n",
      "- If you don't believe us, just read their terms and conditions.\n",
      "\n",
      "---------------------------\n",
      "\n",
      "# 5. > Adults Farm\n",
      "http://80.71.66.8/farm/?aid=760382\n",
      "Girls and Animals Getting Freaky....FREE Lifetime Membership!!\n",
      "\n",
      "# 4. > Sexy Celebes\n",
      "http://80.71.66.8/celebst/?aid=760382\n",
      "Thousands Of XXX Celebes doing it...FREE Lifetime Membership!!\n",
      "\n",
      "# 3. > Play House Porn\n",
      "http://80.71.66.8/mega/?aid=760382\n",
      "Live Feeds From 60 Sites And Web Cams...FREE Lifetime Membership!!\n",
      "\n",
      "# 2. > Asian Sex Fantasies\n",
      "http://80.71.66.8/asian/?aid=760382\n",
      "Japanese Schoolgirls, Live Sex Shows ...FREE Lifetime Membership!!\n",
      "\n",
      "# 1. > Lesbian Lace\n",
      "http://80.71.66.8/lesbian/?aid=760382\n",
      "Girls and Girls Getting Freaky! ...FREE Lifetime Membership!!\n",
      "\n",
      "--------------------------\n",
      "\n",
      "Jennifer Simpson, Miami, FL\n",
      "Your FREE lifetime membership has entertained my boyffriend and I for\n",
      "the last two years!  Your Adult Sites are the best on the net!\n",
      "\n",
      "Joe Morgan Manhattan, NY\n",
      "Your live sex shows and live sex cams are unbelievable. The best part\n",
      "about your porn sites, is that they're absolutely FREE!\n",
      "\n",
      "--------------------------\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Removal Instructions:\n",
      "\n",
      "You have received this advertisement because you have opted in to receive free adult internet\n",
      "offers and specials through our affiliated websites. If you do not wish to receive further emails\n",
      "or have received the email in error you may opt-out of our database here\n",
      "http://80.71.66.8/optout/ . Please allow 24 hours for removal.\n",
      "\n",
      "vonolmosatkirekpups\n"
     ]
    }
   ],
   "source": [
    "print(spam_emails[3].get_content().strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 电子邮件实际上有很多部分，带有图像和附件（他们可以有自己的附件）。查看邮件的各种类型的结构(邮件头，附件，邮件内容类型）：\n",
    "def get_email_structure(email):\n",
    "    if isinstance(email,str):\n",
    "        return email\n",
    "    payload = email.get_payload()\n",
    "    if isinstance(payload,list):\n",
    "        return \"multipart({})\".format(\".\".join([\n",
    "            get_email_structure(sub_email) for sub_email in payload\n",
    "        ]))\n",
    "    else:\n",
    "        return email.get_content_type()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({1: 2, 2: 1, 3: 3, 4: 2, 5: 1})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "a = [1,1,2,3,4,5,3,3,4]\n",
    "b = Counter(a) #求数组中每个数字出现的次数\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "def structures_counter(emails):\n",
    "    structures = Counter()\n",
    "    for email in emails:\n",
    "        structure = get_email_structure(email)\n",
    "        structures[structure] += 1\n",
    "    return structures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('text/plain', 2408),\n",
       " ('multipart(text/plain.application/pgp-signature)', 66),\n",
       " ('multipart(text/plain.text/html)', 8),\n",
       " ('multipart(text/plain.text/plain)', 4),\n",
       " ('multipart(text/plain)', 3),\n",
       " ('multipart(text/plain.application/octet-stream)', 2),\n",
       " ('multipart(text/plain.text/enriched)', 1),\n",
       " ('multipart(text/plain.application/ms-tnef.text/plain)', 1),\n",
       " ('multipart(multipart(text/plain.text/plain.text/plain).application/pgp-signature)',\n",
       "  1),\n",
       " ('multipart(text/plain.video/mng)', 1),\n",
       " ('multipart(text/plain.multipart(text/plain))', 1),\n",
       " ('multipart(text/plain.application/x-pkcs7-signature)', 1),\n",
       " ('multipart(text/plain.multipart(text/plain.text/plain).text/rfc822-headers)',\n",
       "  1),\n",
       " ('multipart(text/plain.multipart(text/plain.text/plain).multipart(multipart(text/plain.application/x-pkcs7-signature)))',\n",
       "  1),\n",
       " ('multipart(text/plain.application/x-java-applet)', 1)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structures_counter(ham_emails).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('text/plain', 218),\n",
       " ('text/html', 183),\n",
       " ('multipart(text/plain.text/html)', 45),\n",
       " ('multipart(text/html)', 20),\n",
       " ('multipart(text/plain)', 19),\n",
       " ('multipart(multipart(text/html))', 5),\n",
       " ('multipart(text/plain.image/jpeg)', 3),\n",
       " ('multipart(text/html.application/octet-stream)', 2),\n",
       " ('multipart(text/plain.application/octet-stream)', 1),\n",
       " ('multipart(text/html.text/plain)', 1),\n",
       " ('multipart(multipart(text/html).application/octet-stream.image/jpeg)', 1),\n",
       " ('multipart(multipart(text/plain.text/html).image/gif)', 1),\n",
       " ('multipart/alternative', 1)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structures_counter(spam_emails).most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return-Path : <12a1mailbot1@web.de>\n",
      "Delivered-To : zzzz@localhost.spamassassin.taint.org\n",
      "Received : from localhost (localhost [127.0.0.1])\tby phobos.labs.spamassassin.taint.org (Postfix) with ESMTP id 136B943C32\tfor <zzzz@localhost>; Thu, 22 Aug 2002 08:17:21 -0400 (EDT)\n",
      "Received : from mail.webnote.net [193.120.211.219]\tby localhost with POP3 (fetchmail-5.9.0)\tfor zzzz@localhost (single-drop); Thu, 22 Aug 2002 13:17:21 +0100 (IST)\n",
      "Received : from dd_it7 ([210.97.77.167])\tby webnote.net (8.9.3/8.9.3) with ESMTP id NAA04623\tfor <zzzz@spamassassin.taint.org>; Thu, 22 Aug 2002 13:09:41 +0100\n",
      "From : 12a1mailbot1@web.de\n",
      "Received : from r-smtp.korea.com - 203.122.2.197 by dd_it7  with Microsoft SMTPSVC(5.5.1775.675.6);\t Sat, 24 Aug 2002 09:42:10 +0900\n",
      "To : dcek1a1@netsgo.com\n",
      "Subject : Life Insurance - Why Pay More?\n",
      "Date : Wed, 21 Aug 2002 20:31:57 -1600\n",
      "MIME-Version : 1.0\n",
      "Message-ID : <0103c1042001882DD_IT7@dd_it7>\n",
      "Content-Type : text/html; charset=\"iso-8859-1\"\n",
      "Content-Transfer-Encoding : quoted-printable\n"
     ]
    }
   ],
   "source": [
    "# 正常邮件更多的是纯文本，垃圾邮件有相当多的HTML\n",
    "# 查看邮件头\n",
    "for header,value in spam_emails[0].items():\n",
    "    print(header,\":\",value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Life Insurance - Why Pay More?'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 里面可能有很多有用的信息，比如发件人的邮件地址（12a1mailbot1@web.de）看起来很可疑\n",
    "# 查看“主题”标题\n",
    "spam_emails[0][\"Subject\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 拆分训练集和测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = np.array(ham_emails + spam_emails)\n",
    "y = np.array([0] * len(ham_emails) + [1] * len(spam_emails))\n",
    "\n",
    "X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.2,random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 需要一个函数来将html转化为纯文本，使用[Beautifulsoup]库，下面的函数首先删除 </head/>部分，然后将所有的</a/>标记转化为单词hyperlink，然后去掉所有的html标记，只留下纯文本。为了可读性，还用一个换行符替换多个换行符，最后取消了html实体（例如&gt；或&nbsp；）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from html import unescape\n",
    "\n",
    "def html_to_plain_text(html):\n",
    "    text = re.sub('<head.*?>.*?</head>','',html,flags=re.M | re.S | re.I)\n",
    "    text = re.sub('<a\\s.*?>','HYPERLINK',text,flags=re.M | re.S | re.I)\n",
    "    text = re.sub('<.*?>','',text,flags=re.M | re.S)\n",
    "    text = re.sub(r'(\\s*\\n)+','\\n',text,flags = re.M | re.S)\n",
    "    return unescape(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<HTML><HEAD><TITLE></TITLE><META http-equiv=\"Content-Type\" content=\"text/html; charset=windows-1252\"><STYLE>A:link {TEX-DECORATION: none}A:active {TEXT-DECORATION: none}A:visited {TEXT-DECORATION: none}A:hover {COLOR: #0033ff; TEXT-DECORATION: underline}</STYLE><META content=\"MSHTML 6.00.2713.1100\" name=\"GENERATOR\"></HEAD>\n",
      "<BODY text=\"#000000\" vLink=\"#0033ff\" link=\"#0033ff\" bgColor=\"#CCCC99\"><TABLE borderColor=\"#660000\" cellSpacing=\"0\" cellPadding=\"0\" border=\"0\" width=\"100%\"><TR><TD bgColor=\"#CCCC99\" valign=\"top\" colspan=\"2\" height=\"27\">\n",
      "<font size=\"6\" face=\"Arial, Helvetica, sans-serif\" color=\"#660000\">\n",
      "<b>OTC</b></font></TD></TR><TR><TD height=\"2\" bgcolor=\"#6a694f\">\n",
      "<font size=\"5\" face=\"Times New Roman, Times, serif\" color=\"#FFFFFF\">\n",
      "<b>&nbsp;Newsletter</b></font></TD><TD height=\"2\" bgcolor=\"#6a694f\"><div align=\"right\"><font color=\"#FFFFFF\">\n",
      "<b>Discover Tomorrow's Winners&nbsp;</b></font></div></TD></TR><TR><TD height=\"25\" colspan=\"2\" bgcolor=\"#CCCC99\"><table width=\"100%\" border=\"0\"  ...\n"
     ]
    }
   ],
   "source": [
    "html_spam_emails = [email for email in X_train[y_train==1] if get_email_structure(email) == 'text/html']\n",
    "sample_html_spam = html_spam_emails[7]\n",
    "print(sample_html_spam.get_content().strip()[:1000],\"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "OTC\n",
      " Newsletter\n",
      "Discover Tomorrow's Winners \n",
      "For Immediate Release\n",
      "Cal-Bay (Stock Symbol: CBYI)\n",
      "Watch for analyst \"Strong Buy Recommendations\" and several advisory newsletters picking CBYI.  CBYI has filed to be traded on the OTCBB, share prices historically INCREASE when companies get listed on this larger trading exchange. CBYI is trading around 25 cents and should skyrocket to $2.66 - $3.25 a share in the near future.\n",
      "Put CBYI on your watch list, acquire a position TODAY.\n",
      "REASONS TO INVEST IN CBYI\n",
      "A profitable company and is on track to beat ALL earnings estimates!\n",
      "One of the FASTEST growing distributors in environmental & safety equipment instruments.\n",
      "Excellent management team, several EXCLUSIVE contracts.  IMPRESSIVE client list including the U.S. Air Force, Anheuser-Busch, Chevron Refining and Mitsubishi Heavy Industries, GE-Energy & Environmental Research.\n",
      "RAPIDLY GROWING INDUSTRY\n",
      "Industry revenues exceed $900 million, estimates indicate that there could be as much as $25 billi ...\n"
     ]
    }
   ],
   "source": [
    "print(html_to_plain_text(sample_html_spam.get_content())[:1000],\"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 编写一个函数，以电子邮件为输入，并以纯文本形式返回其内容，无论其格式是什么\n",
    "def email_to_text(email):\n",
    "    html = None\n",
    "    for part in email.walk():\n",
    "        ctype = part.get_content_type()\n",
    "        if not ctype in (\"text/plain\",\"text/html\"):\n",
    "            continue\n",
    "        try:\n",
    "            content = part.get_content()\n",
    "        except:\n",
    "            content = str(part.get_payload())\n",
    "        if ctype == \"text/plain\":\n",
    "            return content\n",
    "        else:\n",
    "            html = content\n",
    "    if html:\n",
    "        return html_to_plain_text(html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "OTC\n",
      " Newsletter\n",
      "Discover Tomorrow's Winners \n",
      "For Immediate Release\n",
      "Cal-Bay (Stock Symbol: CBYI)\n",
      "Wat ...\n"
     ]
    }
   ],
   "source": [
    "print(email_to_text(sample_html_spam)[:100],'...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computations => comput\n",
      "Computation => comput\n",
      "Computing => comput\n",
      "Computed => comput\n",
      "Compute => comput\n",
      "Compulsive => compuls\n"
     ]
    }
   ],
   "source": [
    "import nltk  # 自然语言工具包\n",
    "from urlextract import URLExtract # 替换邮件中的URL地址\n",
    "try:\n",
    "    stemmer = nltk.PorterStemmer()\n",
    "    for word in (\"Computations\",\"Computation\",\"Computing\",\"Computed\",\"Compute\",\"Compulsive\"):\n",
    "        print(word,\"=>\",stemmer.stem(word))\n",
    "except ImportError:\n",
    "    print(\"Error:stemming requires the NLTK module.\")\n",
    "    stemmer = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 将所有处理整合到一个转换器中，我们将使用他将电子邮件转化为文字，再将文字进行统计计数。注意，我们使用python的'split()'方法将句子拆分为单词，该方法使用空格作为单词边界。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator,TransformerMixin\n",
    "\n",
    "class EmailToWordCounterTransformer(BaseEstimator,TransformerMixin):\n",
    "    def __init__(self,strip_headers=True,lower_case = True,remove_punctuation=True,\n",
    "                 replace_urls=True,replace_numbers=True,stemming=True):\n",
    "        self.strip_headers = strip_headers # 去掉邮件头部\n",
    "        self.lower_case = lower_case # 单词转化为小写\n",
    "        self.remove_punctuation = remove_punctuation #去掉所有的标点符号\n",
    "        self.replace_urls = replace_urls # 替换所有的url地址\n",
    "        self.replace_numbers = replace_numbers # 替换掉所有的数字\n",
    "        self.stemming = stemming # 提取词干\n",
    "    def fit(self,X,y=None):\n",
    "        return self\n",
    "    def transform(self,X,y=None):\n",
    "        X_transformed = []\n",
    "        for email in X:\n",
    "            text = email_to_text(email) or \"\"\n",
    "            if self.lower_case:\n",
    "                text = text.lower()\n",
    "            if self.replace_urls:\n",
    "                extractor = URLExtract()\n",
    "                urls = list(set(extractor.find_urls(text)))\n",
    "                urls.sort(key=lambda url:len(url),reverse=True)\n",
    "                for url in urls:  # 替换url链接为\"URL\"\n",
    "                    text = text.replace(url,\"URL\")\n",
    "            if self.replace_numbers: # 替换数字\n",
    "                text = re.sub(r'\\d+(?:\\.\\d*(?:[eE]\\d+))?','NUMBER',text)\n",
    "            if self.remove_punctuation: \n",
    "                text = re.sub(r'\\W+',' ',text,flags=re.M)\n",
    "            word_counts = Counter(text.split())\n",
    "            if self.stemming and stemmer is not None:\n",
    "                stemmed_word_counts = Counter()\n",
    "                for word,count in word_counts.items():\n",
    "                    stemmed_word = stemmer.stem(word)\n",
    "                    stemmed_word_counts[stemmed_word] += count\n",
    "                word_counts = stemmed_word_counts\n",
    "            X_transformed.append(word_counts)\n",
    "        return np.array(X_transformed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([Counter({'chuck': 1, 'murcko': 1, 'wrote': 1, 'stuff': 1, 'yawn': 1, 'r': 1}),\n",
       "       Counter({'the': 11, 'of': 9, 'and': 8, 'all': 3, 'christian': 3, 'to': 3, 'by': 3, 'jefferson': 2, 'i': 2, 'have': 2, 'superstit': 2, 'one': 2, 'on': 2, 'been': 2, 'ha': 2, 'half': 2, 'rogueri': 2, 'teach': 2, 'jesu': 2, 'some': 1, 'interest': 1, 'quot': 1, 'url': 1, 'thoma': 1, 'examin': 1, 'known': 1, 'word': 1, 'do': 1, 'not': 1, 'find': 1, 'in': 1, 'our': 1, 'particular': 1, 'redeem': 1, 'featur': 1, 'they': 1, 'are': 1, 'alik': 1, 'found': 1, 'fabl': 1, 'mytholog': 1, 'million': 1, 'innoc': 1, 'men': 1, 'women': 1, 'children': 1, 'sinc': 1, 'introduct': 1, 'burnt': 1, 'tortur': 1, 'fine': 1, 'imprison': 1, 'what': 1, 'effect': 1, 'thi': 1, 'coercion': 1, 'make': 1, 'world': 1, 'fool': 1, 'other': 1, 'hypocrit': 1, 'support': 1, 'error': 1, 'over': 1, 'earth': 1, 'six': 1, 'histor': 1, 'american': 1, 'john': 1, 'e': 1, 'remsburg': 1, 'letter': 1, 'william': 1, 'short': 1, 'again': 1, 'becom': 1, 'most': 1, 'pervert': 1, 'system': 1, 'that': 1, 'ever': 1, 'shone': 1, 'man': 1, 'absurd': 1, 'untruth': 1, 'were': 1, 'perpetr': 1, 'upon': 1, 'a': 1, 'larg': 1, 'band': 1, 'dupe': 1, 'import': 1, 'led': 1, 'paul': 1, 'first': 1, 'great': 1, 'corrupt': 1}),\n",
       "       Counter({'url': 4, 's': 3, 'group': 3, 'to': 3, 'in': 2, 'forteana': 2, 'martin': 2, 'an': 2, 'and': 2, 'we': 2, 'is': 2, 'yahoo': 2, 'unsubscrib': 2, 'y': 1, 'adamson': 1, 'wrote': 1, 'for': 1, 'altern': 1, 'rather': 1, 'more': 1, 'factual': 1, 'base': 1, 'rundown': 1, 'on': 1, 'hamza': 1, 'career': 1, 'includ': 1, 'hi': 1, 'belief': 1, 'that': 1, 'all': 1, 'non': 1, 'muslim': 1, 'yemen': 1, 'should': 1, 'be': 1, 'murder': 1, 'outright': 1, 'know': 1, 'how': 1, 'unbias': 1, 'memri': 1, 'don': 1, 't': 1, 'html': 1, 'rob': 1, 'sponsor': 1, 'number': 1, 'dvd': 1, 'free': 1, 'p': 1, 'join': 1, 'now': 1, 'from': 1, 'thi': 1, 'send': 1, 'email': 1, 'egroup': 1, 'com': 1, 'your': 1, 'use': 1, 'of': 1, 'subject': 1})],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在一些邮件上测试转换器\n",
    "X_few = X_train[:3] # 训练集的前三个邮件\n",
    "X_few_wordcounts = EmailToWordCounterTransformer().fit_transform(X_few)\n",
    "X_few_wordcounts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 有了单词计数，我们需要把他们转化成向量。为此，我们将构建另一个转换器，其“fit()”方法将构建词汇表（最常用的词汇有序列表），其“transform()”方法将使用词汇表将单词计数转换为向量——稀疏矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import csr_matrix\n",
    "class WordCounterToVectorTransformer(BaseEstimator,TransformerMixin):\n",
    "    def __init__(self,vocabulary_size = 1000):\n",
    "        self.vocabulary_size = vocabulary_size  # 词汇量\n",
    "    def fit(self,X,y=None):\n",
    "        total_count = Counter()\n",
    "        for word_count in X:\n",
    "            for word,count in word_count.items():\n",
    "                total_count[word] += min(count,10)\n",
    "        most_common = total_count.most_common()[:self.vocabulary_size] #截取最常用的词汇表\n",
    "        self.most_common_ = most_common #赋给成员变量\n",
    "        self.vocabulary_ = {word: index + 1 for index ,(word,count) in enumerate(most_common)}\n",
    "        return self\n",
    "    def transform(self,X,y = None):\n",
    "        rows = [] # 行指标\n",
    "        cols = [] # 列指标\n",
    "        data = [] # 在行指标，列指标下的数字\n",
    "        for row,word_count in enumerate(X):\n",
    "            for word,count in word_count.items():\n",
    "                rows.append(row) # 训练集 实例个数\n",
    "                cols.append(self.vocabulary_.get(word,0)) # 取得单词在词汇表中的索引位置，0代表未出现在词汇表中\n",
    "                data.append(count)\n",
    "        return csr_matrix((data,(rows,cols)),shape = (len(X),self.vocabulary_size +1 )) #输出稀疏矩阵 +1因为第一列要显示未出现过的词汇数\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 11)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_transformer = WordCounterToVectorTransformer(vocabulary_size = 10)\n",
    "X_few_vectors = vocab_transformer.fit_transform(X_few_wordcounts)\n",
    "X_few_vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [99, 11,  9,  8,  3,  1,  3,  1,  3,  2,  3],\n",
       "       [67,  0,  1,  2,  3,  4,  1,  2,  0,  1,  0]], dtype=int32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_few_vectors.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第三行第一列中的67表示三封电子邮件中包含67不属于词汇表的单词。后面的0表示‘the’在此电子邮件中没有出现，后面的1表示词汇表中‘of’单词在此电子邮件中出现一次，后面的2表示‘and’出现两次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'of': 2,\n",
       " 'and': 3,\n",
       " 'to': 4,\n",
       " 'url': 5,\n",
       " 'all': 6,\n",
       " 'in': 7,\n",
       " 'christian': 8,\n",
       " 'on': 9,\n",
       " 'by': 10}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_transformer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练第一个垃圾邮件分类器，首先转换整个数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "preprocess_pipeline = Pipeline([\n",
    "    ('email_to_wordcounter',EmailToWordCounterTransformer()),\n",
    "    ('wordcount_to_vector',WordCounterToVectorTransformer()),\n",
    "])\n",
    "\n",
    "X_train_transformed = preprocess_pipeline.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  ................................................................\n",
      "[CV] .................................... , score=0.981, total=   0.1s\n",
      "[CV]  ................................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .................................... , score=0.981, total=   0.2s\n",
      "[CV]  ................................................................\n",
      "[CV] .................................... , score=0.991, total=   0.2s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.2s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    0.3s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9845833333333333"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "log_clf = LogisticRegression(solver=\"liblinear\",random_state=42) # 逻辑回归分类器\n",
    "score = cross_val_score(log_clf,X_train_transformed,y_train,cv=3,verbose=3)\n",
    "score.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 得到的分数超过98.4%，可以尝试多个模型，选择最好的模型，并使用交叉验证对他们进行微调。在测试集上得到的精度/召回率："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "精度：93.94%\n",
      "召回率：93.94%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score,recall_score\n",
    "\n",
    "X_test_transformed = preprocess_pipeline.transform(X_test)\n",
    "\n",
    "log_clf = LogisticRegression(solver = 'liblinear',random_state=42)\n",
    "log_clf.fit(X_train_transformed,y_train)\n",
    "\n",
    "y_pred = log_clf.predict(X_test_transformed)\n",
    "\n",
    "print(\"精度：{:.2f}%\".format(100 * precision_score(y_test,y_pred)))\n",
    "print(\"召回率：{:.2f}%\".format(100 * precision_score(y_test,y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "   1. 加载数据并纵观数据大局\n",
    "   2. 获取邮件的组成机构\n",
    "   3. 对邮件结构类型进行分析，发现垃圾邮件大多有HTML结构\n",
    "   4. 数据清洗，定义email对象中的HTML转化成纯文本方法\n",
    "   5. 对数据集拆分成训练集和测试集\n",
    "   6. 数据处理转换，对邮件的文本内容进行分词，通过nltk进行词干提取，汇总垃圾邮件中频繁出现的词汇的计数统计，对所有邮件统计出一个词汇表\n",
    "   7. 通过此词汇表和邮件单词计数统计，将单词技术转化成向量矩阵\n",
    "   8. 把数据清洗和数据处理封装成两个转换器\n",
    "   9. 通过流水线来自动化处理数据\n",
    "   10. 使用逻辑回归线性分类器进行模型训练\n",
    "   11. 使用交叉验证\n",
    "   12. 在测试集上得到精度和召回率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
